fix(operator): add 100MB memory limit to binary file scan to prevent OOM by roshiiiz · Pull Request #6811 · apache/texera

roshiiiz · 2026-07-23T00:17:26Z

What changes were proposed in this PR?

This PR introduces a strict 100 MB memory allocation limit when scanning files using the standard binary attribute type.

Why is it needed?
Previously, when users mistakenly attempted to read massive files (e.g., >8GB) using the binary attribute type (instead of the streaming large binary type), the ByteArrayOutputStream would attempt to allocate the entire file into a single contiguous Java byte array. Since standard byte[] arrays max out at ~2.14GB, this inherently fails. Worse, on standard developer laptops or lower-memory deployments, this massive allocation attempt triggered immediate IllegalArgumentExceptions or severe JVM Garbage Collection death spirals, causing the computing-unit-master to lock up or crash entirely without reporting a user-friendly error to the frontend UI.

What was changed:

Added a MAX_SAFE_SIZE limit of 100 MB to FileScanUtils.scala.
During the binary scan loop, if the byte stream exceeds 100 MB, it is immediately intercepted and aborted safely.
Throws a clean, user-friendly RuntimeException directly to the frontend directing the user to use the large binary attribute type instead for massive files, completely preventing the JVM from thrashing or freezing.

Any related issues, documentation, discussions?

Closes #3271

How was this PR tested?

Manual Verification:

Uploaded an 8.9 GB CSV test file to the workspace.
Created a workflow with the File Scan operator.
Configured the operator to use the binary attribute type (which intentionally attempts to load the entire file into memory).
Ran the workflow.
Result: The workflow aborted instantly (in ~3-4 seconds) the moment it crossed the 100MB threshold. It safely threw the custom user-friendly error message in the UI and immediately cleaned up the worker thread without any memory leaks or GC pauses.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Antigravity (DeepMind)

Changed it to 100 Mb instead of 1gb afterwards

…nfigured operators

- reformat the over-100-column require() in AsterixDBSourceOpDesc so scalafmtCheckAll passes - align the scan-source missing-fileName error in both LogicalPlan copies (amber + workflow-compiling-service) with the new 'No file selected...' wording that the updated specs expect - WorkflowCompiler: keep the first error recorded per operator so the root-cause resolution failure (e.g. FileNotFoundException with the bad path) is not overwritten by follow-on schema-propagation failures on the same operator - update workflow-operator DescSpecs that pinned the old contracts: sourceSchema() returning null before configuration (CSVOld/JSONL/MySQL/PostgreSQL/AsterixDB/SQL) and ArrowSourceOpDesc.inferSchema throwing a bare IOException

github-actions · 2026-07-23T00:17:44Z

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

Contributors with relevant context: @aglinxinyuan
You can notify them by mentioning @aglinxinyuan in a comment.

codecov-commenter · 2026-07-23T00:20:43Z

Codecov Report

❌ Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.57%. Comparing base (b4d1865) to head (65e7f2d).

Files with missing lines	Patch %	Lines
...mber/operator/source/scan/file/FileScanUtils.scala	73.33%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6811      +/-   ##
============================================
- Coverage     78.69%   78.57%   -0.13%     
+ Complexity     3731     3728       -3     
============================================
  Files          1161     1161              
  Lines         46066    46081      +15     
  Branches       5106     5110       +4     
============================================
- Hits          36253    36207      -46     
- Misses         8208     8260      +52     
- Partials       1605     1614       +9

Flag	Coverage Δ		*Carryforward flag
access-control-service	`70.00% <ø> (ø)`
agent-service	`76.76% <ø> (ø)`		Carriedforward from 03a0e3a
amber	`71.50% <73.33%> (-0.03%)`	⬇️
computing-unit-managing-service	`20.49% <ø> (ø)`
config-service	`66.66% <ø> (ø)`
file-service	`67.21% <ø> (ø)`
frontend	`82.64% <ø> (-0.27%)`	⬇️	Carriedforward from 03a0e3a
notebook-migration-service	`78.94% <ø> (ø)`
pyamber	`95.40% <ø> (+0.04%)`	⬆️	Carriedforward from 03a0e3a
workflow-compiling-service	`55.14% <ø> (ø)`

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-07-23T00:23:59Z

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 5 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main b4d1865 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	357	0.218	26,621/41,410/41,410 us	🔴 +39.0% / 🔴 +152.9%
⚪	bs=100 sw=10 sl=64	798	0.487	126,190/149,642/149,642 us	⚪ within ±5% / 🔴 +36.9%
⚪	bs=1000 sw=10 sl=64	914	0.558	1,095,781/1,127,775/1,127,775 us	⚪ within ±5% / 🔴 +8.8%

Baseline details

Latest main b4d1865 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	357 tuples/sec	430 tuples/sec	759.72 tuples/sec	-17.0%	-53.0%
bs=10 sw=10 sl=64	MB/s	0.218 MB/s	0.262 MB/s	0.464 MB/s	-16.8%	-53.0%
bs=10 sw=10 sl=64	p50	26,621 us	22,271 us	12,653 us	+19.5%	+110.4%
bs=10 sw=10 sl=64	p95	41,410 us	29,800 us	16,371 us	+39.0%	+152.9%
bs=10 sw=10 sl=64	p99	41,410 us	29,800 us	19,941 us	+39.0%	+107.7%
bs=100 sw=10 sl=64	throughput	798 tuples/sec	814 tuples/sec	966.78 tuples/sec	-2.0%	-17.5%
bs=100 sw=10 sl=64	MB/s	0.487 MB/s	0.497 MB/s	0.59 MB/s	-2.0%	-17.5%
bs=100 sw=10 sl=64	p50	126,190 us	121,252 us	103,654 us	+4.1%	+21.7%
bs=100 sw=10 sl=64	p95	149,642 us	146,292 us	109,308 us	+2.3%	+36.9%
bs=100 sw=10 sl=64	p99	149,642 us	146,292 us	116,369 us	+2.3%	+28.6%
bs=1000 sw=10 sl=64	throughput	914 tuples/sec	941 tuples/sec	997.95 tuples/sec	-2.9%	-8.4%
bs=1000 sw=10 sl=64	MB/s	0.558 MB/s	0.574 MB/s	0.609 MB/s	-2.8%	-8.4%
bs=1000 sw=10 sl=64	p50	1,095,781 us	1,061,660 us	1,007,348 us	+3.2%	+8.8%
bs=1000 sw=10 sl=64	p95	1,127,775 us	1,104,414 us	1,050,149 us	+2.1%	+7.4%
bs=1000 sw=10 sl=64	p99	1,127,775 us	1,104,414 us	1,079,497 us	+2.1%	+4.5%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,560.01,200,128000,357,0.218,26620.63,41409.75,41409.75
1,100,10,64,20,2507.69,2000,1280000,798,0.487,126190.35,149642.17,149642.17
2,1000,10,64,20,21892.72,20000,12800000,914,0.558,1095781.32,1127775.45,1127775.45

roshiiiz · 2026-07-23T01:33:27Z

If the Java limit is about 2 GB, should we set it to 2 GB instead of 100 MB? What if I want to upload a 200MB as binary?

Great question! While the theoretical Java byte limit is 2GB, ByteArrayOutputStream doubles its internal buffer capacity every time it resizes. This means reading a 1GB file actually requires >3GB of free heap space just to perform the array copy. On standard local deployments, this instantly triggers OutOfMemoryErrors or severe GC death spirals that crash the worker JVM before it ever hits the 2GB limit.

The 100MB limit was chosen as a conservative safety net to guarantee the worker never crashes, while actively nudging users to select the large binary attribute type instead for bigger files (since large binary streams the data chunk-by-chunk and creates zero heap pressure).

However, I am completely open to bumping this limit to 500MB if you feel that's a better threshold for standard use cases! Let me know what limit you'd prefer and I'll gladly update the PR.

kunwp1 · 2026-07-24T14:58:10Z

Do you think producing OOM is a bug? I think it's natural to produce OOM error if the operator doesn't support reading more than 2GB file. I am okay if the direction is to make the error message user-friendly and guide to using large-binary type but I don't think introducing a threshold is a good solution.

roshiiiz · 2026-07-24T16:37:31Z

Do you think producing OOM is a bug? I think it's natural to produce OOM error if the operator doesn't support reading more than 2GB file. I am okay if the direction is to make the error message user-friendly and guide to using large-binary type but I don't think introducing a threshold is a good solution.

I completely understand your point, and I agree that an OOM is natural when hitting that 2GB limit.
The main reason I initially leaned toward a threshold was actually to solve a user experience issue. The original use case that sparked this involved someone uploading a 9GB file. They ended up waiting a really long time for it to process, only for the whole thing to crash at the very end with a generic error. My goal was to introduce a way to 'fail fast' so a new user unaware of the limit doesn't waste 30 minutes waiting on a 10GB file just to get an OOM.

kunwp1 · 2026-07-24T18:49:28Z

If you want a faster fail path, then I wonder why we have to run the operator to check the file size. Did you consider a way to check the file size before running the operator? Also, exceeding 100MB doesn't necessarily mean the operator will cause an OOM. Can you reconsider the idea?

roshiiiz · 2026-07-24T19:18:51Z

If you want a faster fail path, then I wonder why we have to run the operator to check the file size. Did you consider a way to check the file size before running the operator? Also, exceeding 100MB doesn't necessarily mean the operator will cause an OOM. Can you reconsider the idea?

That is a great suggestion, but checking the file size upfront introduces some edge cases. For instance, if a user uploads a .zip file, we cannot determine the uncompressed size of the extracted entries until the operator is actively streaming and unzipping them on the fly. (I might be overthinking this as well)

However, I completely agree that your original suggestion is 100% the cleanest and best approach. I will go ahead and remove the threshold entirely. I'll update the PR to let it hit the Java array limit, and we can simply catch the resulting OutOfMemoryError or IllegalArgumentException and wrap it in user-friendly large-binary hint.

Thanks for the feedback I will update the PR with this approach shortly.

kunwp1 · 2026-07-24T19:28:24Z

Sounds good! I haven't thought about a zip file but your opinion makes sense that it's not possible to cover all the cases and check the file size beforehand. You can go ahead with the fix.

aglinxinyuan · 2026-07-24T20:04:08Z

If the Java limit is about 2 GB, should we set it to 2 GB instead of 100 MB? What if I want to upload a 200MB as binary?

Great question! While the theoretical Java byte limit is 2GB, ByteArrayOutputStream doubles its internal buffer capacity every time it resizes. This means reading a 1GB file actually requires >3GB of free heap space just to perform the array copy. On standard local deployments, this instantly triggers OutOfMemoryErrors or severe GC death spirals that crash the worker JVM before it ever hits the 2GB limit.

The 100MB limit was chosen as a conservative safety net to guarantee the worker never crashes, while actively nudging users to select the large binary attribute type instead for bigger files (since large binary streams the data chunk-by-chunk and creates zero heap pressure).

However, I am completely open to bumping this limit to 500MB if you feel that's a better threshold for standard use cases! Let me know what limit you'd prefer and I'll gladly update the PR.

Do you have some data to support this? For example, can you test the maximum possible file size for this operator? We want to set this limit to upper bound instead of lower bound.

chenlica · 2026-07-26T05:41:18Z

@roshiiiz Can you confirm #3721 (about login) is related to this PR?

Also if we want to go with this design, it will be good to make this size configurable by the admins.

roshiiiz · 2026-07-26T13:26:00Z

If the Java limit is about 2 GB, should we set it to 2 GB instead of 100 MB? What if I want to upload a 200MB as binary?

Great question! While the theoretical Java byte limit is 2GB, ByteArrayOutputStream doubles its internal buffer capacity every time it resizes. This means reading a 1GB file actually requires >3GB of free heap space just to perform the array copy. On standard local deployments, this instantly triggers OutOfMemoryErrors or severe GC death spirals that crash the worker JVM before it ever hits the 2GB limit.
The 100MB limit was chosen as a conservative safety net to guarantee the worker never crashes, while actively nudging users to select the large binary attribute type instead for bigger files (since large binary streams the data chunk-by-chunk and creates zero heap pressure).
However, I am completely open to bumping this limit to 500MB if you feel that's a better threshold for standard use cases! Let me know what limit you'd prefer and I'll gladly update the PR.

Do you have some data to support this? For example, can you test the maximum possible file size for this operator? We want to set this limit to upper bound instead of lower bound.

Thanks for the feedback. I actually had a discussion with @kunwp1 in the thread above relating somewhat to this exact concern. We agreed that instead of trying to find the perfect artificial upper bound, it is much cleaner to just remove the hardcoded threshold entirely.

I am updating the PR to let the operator naturally hit the Java array limit on its own, and we can simply catch the resulting OutOfMemoryError or IllegalArgumentException and wrap it in the user-friendly large-binary hint.

roshiiiz · 2026-07-26T13:31:43Z

@roshiiiz Can you confirm #3721 (about login) is related to this PR?

Also if we want to go with this design, it will be good to make this size configurable by the admins.

Nope, I can confirm this PR is completely unrelated to #3721. My frontend changes just strip technical exception prefixes from operator warnings in the global error handler so it doesn't touch the login/registration notification durations at all.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…n to prevent OOM

chenlica · 2026-07-26T22:42:25Z

Closes #3721

@roshiiiz Can you confirm #3721 (about login) is related to this PR?
Also if we want to go with this design, it will be good to make this size configurable by the admins.

Nope, I can confirm this PR is completely unrelated to #3721. My frontend changes just strip technical exception prefixes from operator warnings in the global error handler so it doesn't touch the login/registration notification durations at all.

The description mentioned #3721. Please fix it.

roshiiiz and others added 12 commits July 9, 2026 13:23

fix(frontend): improve static error msgs to be user-friendly for unco…

a19030e

…nfigured operators

Merge branch 'main' into fix/user-friendly-error-messages

8af10dd

address copilot PR review comments

74c6895

Merge branch 'main' into fix/user-friendly-error-messages

b61adda

address additional copilot PR review comments

5879ca6

address copilot PR review comments (round 3)

35821a2

address copilot PR review comments (round 4)

82b5912

address copilot PR review comments (round 5)

14282f8

Merge branch 'main' into fix/user-friendly-error-messages

346b114

address copilot PR review comments (round 6)

2f94fe4

fix formatting issues

faa965a

github-actions Bot assigned roshiiiz Jul 23, 2026

github-actions Bot added engine fix frontend Changes related to the frontend GUI common platform Non-amber Scala service paths labels Jul 23, 2026

github-actions Bot removed engine frontend Changes related to the frontend GUI platform Non-amber Scala service paths labels Jul 23, 2026

roshiiiz force-pushed the fix/file-scan-memory-limit branch from 2e58992 to 796f392 Compare July 23, 2026 00:26

github-actions Bot added engine frontend Changes related to the frontend GUI platform Non-amber Scala service paths and removed engine frontend Changes related to the frontend GUI labels Jul 23, 2026

xuang7 added the release/v1.2 back porting to release/v1.2 label Jul 23, 2026

roshiiiz force-pushed the fix/file-scan-memory-limit branch from 509246d to 0386079 Compare July 26, 2026 13:42

github-actions Bot added engine frontend Changes related to the frontend GUI platform Non-amber Scala service paths and removed engine frontend Changes related to the frontend GUI platform Non-amber Scala service paths labels Jul 26, 2026

roshiiiz requested a review from Copilot July 26, 2026 13:50

Copilot AI reviewed Jul 26, 2026

fix(operator): gracefully handle memory limits during binary file sca…

861c761

…n to prevent OOM

roshiiiz force-pushed the fix/file-scan-memory-limit branch from c0793ce to 861c761 Compare July 26, 2026 14:33

github-actions Bot added engine frontend Changes related to the frontend GUI platform Non-amber Scala service paths labels Jul 26, 2026

Merge branch 'main' into fix/file-scan-memory-limit

65e7f2d

github-actions Bot removed engine frontend Changes related to the frontend GUI platform Non-amber Scala service paths labels Jul 26, 2026

Uh oh!

Conversation

roshiiiz commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Reviewer Suggestions

Uh oh!

codecov-commenter commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Benchmark changes need a look

Uh oh!

roshiiiz commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kunwp1 commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roshiiiz commented Jul 24, 2026

Uh oh!

kunwp1 commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roshiiiz commented Jul 24, 2026

Uh oh!

kunwp1 commented Jul 24, 2026

Uh oh!

aglinxinyuan commented Jul 24, 2026

Uh oh!

chenlica commented Jul 26, 2026

Uh oh!

roshiiiz commented Jul 26, 2026

Uh oh!

roshiiiz commented Jul 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chenlica commented Jul 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

roshiiiz commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading

codecov-commenter commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading

roshiiiz commented Jul 23, 2026 •

edited

Loading

kunwp1 commented Jul 24, 2026 •

edited

Loading

kunwp1 commented Jul 24, 2026 •

edited

Loading